Full text search (FTS) systems search for relevant documents based on key words entered by a system user. The user enters a set of terms, referred to as tokens, and the FTS system finds documents containing all of the terms in the set. In order to support queries efficiently, the FTS system typically uses inverted indexes. For example, Lucene (described at http://lucene.apache.org/) and SQLite's FTS module (described at http://www.sqlite.org/cvstrac/wiki?p=FtsUsage) are both FTS systems that use inverted indexes.
An inverted index assigns a set of document identifiers to each token. The document identifiers are associated with documents that include the token at least once. Upon receiving a search request, the FTS system selects the set of document identifiers for each token in the request and then compares the document sets to each other. If a document identifier is contained in all document sets, the FTS system provides the document identifier in a result set of all identifiers contained in all document sets.
From a logical point of view, the inverted index can be regarded as a relation InvertedIndex(Term, DocID) with a combined index on Term and DocId. If a user of the FTS system enters the token “Neuschwanstein,” the inverted index allows the FTS system to efficiently execute queries such as the following query.
SELECT DocID FROM InvertedIndex WHERE Term=‘Neuschwanstein’ However, if the user misspells “Neuschwanstein,” SQLite's FTS system will not find any relevant documents. That is because SQLite's FTS system does not support fault-tolerant (or fuzzy) searching.
Lucene's FTS system does support fuzzy search. However, Lucene performs a fuzzy search in two steps. First, Lucene searches for tokens stored in the database that are similar to the query tokens. To determine if tokens are similar, Lucene computes an edit distance (also referred to as a Levenshtein Distance) from the query tokens to the tokens stored in the database. Second, Lucene uses the similar tokens it finds as new query tokens to retrieve relevant documents. This two-step process may result in severe performance problems.